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^ Abstract 

Q^ In this paper we are interested in the Maximum Likehhood Estimator 

^ (MLE) of the vector parameter of an autoregressive process of order -p with 

regular stationary Gaussian noise. We exhibit the large sample asymptotical 
pg properties of the MLE under very mild conditions. Simulations are done for 

fractional Gaussian noise (fGn), autoregressive noise (AR(1)) and moving 
r I average noise (MA(1)). 
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1 Statement of the problem 
1.1 Introduction 



The problem of parametric estimation in classical autoregressive (AR) models gen- 
qn^ erated by white noises noises has been studied for decades. In particular, for such 

^ autoregressive models of order 1 (AR(1)) consistency and many other asymptotic 

properties (distribution, bias, quadratic error) of the Maximum Likelihood Esti- 
mator (MLE) have been completely analyzed in all possible cases: stable, unstable 
O and explosive (see, e.g.^ [21 HI EHl EB ES EE]). Concerning autoregressive models 

^ of order p (AR(p)) with white noises, the results about the asymptotic behavior of 

L| the MLE are less exhaustive but there are still many contributions in the literature 

(see, e.^.,[lEl[ni[iaElE|). 

In the past thirty years numerous papers have been devoted to the statistical 
cd analysis of AR processes which may represent long memory phenomenons as en- 

countered in various fields as econometrics |9], hydrology [13] or biology ^iSj. Of 
course the relevant models exit from the white noise frame evoked above and they 
involve more or less specific structures of dependence in the perturbations (see, 
e.5'., [H 13 El Uni ESI EH] for contributions and other references). 



General conditions under which the MLE is consistent and asymptotically nor- 
mal for stationary sequences have been given in [23j- In order to apply this result, 
it would be necessary to study the second derivatives of the covariance matrix 
of the observation sample {Xi, . . . ,Xn). To avoid this difficulty, some authors 
followed an other approach suggested by Whittle ^ (which is not MLE) for sta- 
tionary sequences. But even in autoregressive models of order 1 as soon as l-i^l > 1, 
the process is not stationary anymore and it is not possible to apply theorems in 
[7] to deduce estimator properties. 

In the present paper we deal with an AR(p) model generated by an arbitrary 
regular stationary Gaussian noise. We exhibit an explicit formula for the MLE of 
the parameter and we analyze its asymptotic properties. 

1.2 Statement of the problem 

We consider an AR(p) process (X„, n > 1) defined by the recursion 

p 

X^ = J2^^Xn-^+Cn, n > 1, X, = 0, r = 0, -1, . . . , -(p - 1), (1) 

where ^ = (^„, n G Z) is a centered regular stationary Gaussian sequence, i.e. 

r \\nf^iX)\dX<oo, (2) 

where /^(A) is the spectral density of ^. We suppose that the covariance c = 
{c{m,n), m,n> 1), where 

E^rn^n = c{m, n) = p{\n-m\), p(0) = 1, (3) 

is positive defined. 

For a fixed value of the parameter {} = {{}i, . . . , 'dp) G MP, let P^ denote the 
probability measure induced by X^'^\ Let €{1!}, X^^') be the likelihood function 
defined by the Radon-Nikodym derivative of P^ with respect to the Lebesgue 
measure. Our goal is to study the large sample asymptotical properties of the 
Maximum Likelihood Estimator (MLE) -t^N of ^ based on the observation sample 
XW = (Xi,...,X^): 

^N = sup £(^, X(^)). (4) 

At first, preparing for the analysis of the consistency (or strong consistency) of 
i^AT and its limit distribution we transform our observation model into an "equiv- 
alent" model with independent Gaussian noises. This allows to write explicitly 
the MLE and actually, the difference between 'Sn and the real value i) appears as 



the product of a martingale by the inverse of its bracket process. Then we can 
use Laplace transforms computations to prove the asymptotical properties of the 
MLE. 

The paper is organized as follows. Section [2] contains theoretical results and 
simulations. Sections [3] and |4] are devoted to preliminaries and auxiliary results. 
The proofs of the main results are presented in Section |5j 

Acknowledgments We would like to thank Alain Le Breton for very fruitful 
discussions and his interest for this work. 



2 Results and illustrations 

2.1 Results 

We define the p x p companion matrix Aq and the vector b eW as follows: 
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Let r{{}) be the spectral radius of Aq. The following results hold: 
Theorem 2.1. Let p > 1 and the parameter set be: 

e = {^ G M^ I r(^) < 1} . 
The MLE {}n is consistent, i.e., for any -(9 G O and z/ > 0, 



lim P^ 

Af-s>oo 



^N-^ 



> V 



0, 



and asymptotically normal 



N{d^-d^ ^Ar(0, X-^(^)), 



where 1,{if) is the unique solution of the Lyapounov equation: 

X{d) = AQX{d)Al^hh\ 
for Aq and h defined in dsl). 



(5) 



(6) 



(7) 



(9) 



Moreover we have the convergence of the moments: for any f? G and g > 



hm 



E, 



N{^N-^ 



-E\\ri\ 



0, 



(10) 



where || || denotes the Euchdian norm on M^ and 77 is a zero mean Gaussian random 
vector with covariance matrix X{'d)~^. 

Remark 2.1. It is worth to emphasize that the asymptotic covariance X~^{{}) is 
actually the same as in the standard case where (^„) is a white noise, (c/. [?]). 



In the case p = 1 we can strengthen the assertions of Theorem 2.1 In particu- 
lar, the strong consistency and uniform convergence on compacts of the moments 
hold. 

Theorem 2.2. Let p = 1 and the parameter set be G = M. The MLE -t^^ is strongly 
consistent, i.e. for any ?? G 6 



lim i) 



N 



^ 



a.s.. 



Ill 



Moreover, i^N is uniformly consistent and satisfies the uniform convergence of the 
moments on compacts K C (—1, 1), i.e. for any z/ > : 



lim sup P^ < 



^N-^ 



> V 







and for any g > :, 



lim sup 



N^oo 



i9eK 



E, 



N ^ 



'N 



^ 



E 



:i2i 



(13) 



where 77 ~ A/'(0, l-d"^). 

Remark 2.2. It is worth mentioning that condition ^ can be rewritten in terms 
of the covariance function p : p{n) ~ n"", a > 0. 



2.2 Simulations 

In this section we present for p = 1 three illustrations of the behavior of the MLE 
corresponding to noises which are MA(1), AR(1) and fGn. 

Moving average noise MA(1) Here we consider MA(1) noises where 

1 



6 



n+l 



VT 



[En+i + aen), n>l, 



a" 



where (£„,n > 1) is a sequence of i.i.d. zero-mean standard Gaussian variables. 
Then the covariance function is given by 

p{\n - ni\) = l{\n~m\=0} + ^ ^ l{|n-m|=l}- 

Condition ^ is fulfilled for \a\ < 1. 



Autoregressive noise (AR(1)) Here we consider stationary autoregressive 
AR(1) noises where 

in+i = y/l - a%„+i + aCn, n>l, 

where {en,n > 1) is a sequence of i.i.d. zero-mean standard Gaussian variables. 
Then the covariance function is 

p{\n-m\) = al'^-"^l. 

Condition ^ is fulfilled for \a\ < 1. 

Fractional Gaussian noise fGn Here the covariance function of {C,n) is 

p{\m - n|) = - (|m - n + l]^-^ - 2|m - n|^^ + \m - n - 1|^^) , 

for a known Hurst exponent H e (0, 1). For simulation of the fGn we use Wood 
and Chan method (see [2Z])- The explicit formula for the spectral density of fGn 
sequence has been exhibited in [22] ■ Condition (|2]) is fulfilled for any H e (0, 1). 



On Figure [T] we can see that in conformity with Theorem 2.2 in the three 



cases the MLE is asymptotically normal with the same limiting variance as in the 
classical i.i.d. case. 



3 Preliminaries 

3.1 Stationary Gaussian sequences 

We begin with some well known properties of a stationary scalar Gaussian sequence 
^ = (^n)n>i- We denote by (crn£n)n>i ihe innovation type sequence of ^ defined by 

0"l£l = i\, 0-n£n = ^n- E(^„ | 6, • • • , U-l), n > 2, 

where £„ ~ A/'(0,1), n > 1 are independent. It follows from the Theorem of 
Normal Correlation ([IE], Theorem 13.1) that there exists a deterministic kernel 
denoted by k{n,m), n > 1, m < n, such that 



o-n£n = '^k{n,m)^rn, k{n,n) = 1. (14) 

m=l 

In the sequel, for n > 1, we denote by /3„_i the partial correlation coefficient 

- A;(n, 1) = /3„_i, n > 1. (15) 
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Figure 1: Asymptotical normality N = 2000 for the MLE in different cases by 
Monte-Carlo simulation of M = 10000 independent replications for AR(1) noises 
(top left) and MA noises (top right), both for a = 0.4 and i) = 0.2, and fGn noises 
for H = 0.2 (bottom left) and for H = 0.8 (bottom right) both for t9 = 0.8. 

The following relations between k{-, ■), the covariance function p(-) defined by 
([3]), the sequence of partial correlation coefficients (/3n)„>i and the variances of 
innovations {o'n)n>i hold (see Levinson-Durbin algorithm [6]) 



n-l 



^l=ll^^-Pl), n>2, ai = l. 



m=l 



y k{n, m)p{m) = /3„cr, 



2 



(16) 



(17) 



m=l 



k{n + 1, n + 1 — m) = k{n, n — m) — Pnk{n, m). 



Since we assume the positive definiteness of the covariance c(-, ■), there also exists 
an inverse deterministic kernel K = {K{n, m), n > 1, m < n) such that 

n 

^„ = ^ K{n, m)am€m- (19) 

m=l 

Remark 3.1. Actually, kernels k and K are nothing but the ingredients of the 
Choleski decomposition of covariance and inverse of covariance matrices. Namely, 

T-' = k^D-'k: and r„ = K^D^iT^, 

where r„ = {{p{\i — j\))) , kn and Kn are n x n lower triangular matrices with 
ones as diagonal entries and k{i,j) and K{i,j) as subdiagonal entries respectively 
and Dn is an n x n diagonal matrix with af as diagonal entries. Here * denotes 
the transposition. 

Remark 3.2. It is worth mentioning that condition (pi) implies that 

5^/3^ <oo. (20) 

n>l 

Indeed, for every regular stationary Gaussian sequence ^ = (^„, ra G Z), there 
exists a sequence of i.i.d M{0, 1) random variables (£„, n E Z) and a sequence of 
real numbers a^, k > with ao 7^ such that: 

00 

A:=0 

and for all n G Z the cr-algebra generated by (^fc)-oo<fc<n coincides with the a- 
algebra generated by (?fc)-oo<fc<n- 

Note that the variance a"^ of the innovations is also the one step predicting 
error and the following equalities hold thanks to the stationarity of ^: 

n-l 
m=l 



hm E (a - E(en|6, ■ ■ ■ ^n-i)Y = lim E (^o - £(61^-1, ■ ■ ■ ^-n+i)) 

n— >oo n— >oo 

= E (^0 - E(eo|es, s < -1))' = E (^0 - E(eok., s < -1))' = al>0 



which implies (20). 



3.2 Model Transformation 

As usual, for the first step we extend the dimension of the observations in or- 
der to work with a first order autoregression in MP. Namely, let Yn, n > 1, be 
Yn = (X„, Xn-i, . . . , X„_(p_i))* then Y = (Yn, n > 1) satisfies the first order 
autoregressive equation: 

F„ = Aoy„_i + 6^, n>l, Fo = Opxi, (21) 

where Aq and b are defined in ^. For the second step we take an appropriate 
linear transformation of Y in order to have i.i.d. noises in the corresponding 
observations. For this goal let us introduce the process Z = {Zn, n > 1) such that 

n 

Zn = Y, Hn, m)Ym, n>l, (22) 

m=l 



where k = {k{n, m), n >1, m < n) is the kernel appearing in (14). Since we have 
also 

n 

Yn = Y,K{n,m)Zm, (23) 

m=l 

where K = {K(n, m), n > 1, m < n) is the inverse kernel of k (see ([l9|)), the 
filtration of Z coincides with the filtration of Y (and also the filtration of X). 
Actually, it was shown in [3] that Z can be considered as the first component of 
a 2p dimensional AR{1) process ( = (C„, n > 1) governed by i.i.d. noises. More 
precisely, the process C = iCn, n > 1) defined by : 



/ Zn \ 



S)»i 



n-l 



\r=l / 



is a 2p-dimensional Markovian process which satisfies the following equation: 

Cn = A„_iC„_i + lanSn, U > 1, (q = 02pxl, (24) 

where 

^"=(5H 1^°)- «=(o ' )• P^) 

and {Sn, n > 1) are i.i.d. zero mean standard Gaussian variables. Now the ini- 
tial estimation problem is replaced by the problem of estimation of the unknown 
parameter i} from the observations oi ( = (C„, n > 1). 



3.3 Maximum Likelihood Estimator 



It follows directly from equation (24) that the log-likelihood function is nothing 
but: 






n=l ^ n / ^^^ 



and that the maximum likelihood estimator -dN is: 

^^= L -2 ■ L -2 • (26) 

\n=l » / Vn=l " / 

Then we can write 

79^-^ = ((M)^)-^-M^, (27) 

where 

M^ = f; ^^-i^.„ , (M)^ = f: ""-^^"-f -^""-\ (28) 

n=l n=l " 



AClpxp 



(29) 



with 

Note that (M„, n > 1) is a martingale and {{M)n, n > 1) is its bracket process. 
Remark 3.3. It is worth mentioning that in the classical i.i.d. case, i.e., when 



/9„ = 0, n > 1, Mat and {M)n in equations (27)- (28) reduce to: 



N N 

n=l n=l 

Of course, under the condition r{d) < 1 due to the law of the large numbers and 
the central limit theorem for martingales the following convergences hold: 

P^- lim 1(M)^=X(^), ^M;v^Ar(0,X(^)), (30) 



where X(^) is the unique solution of the Lyapounov equation ^. This implies 
immediately the consistency and the asymptotic normality of the MLE. 



4 Auxiliary results 



Actually, the proof of Theorems 2.1 - 2.2 is crucially based on the asymptotic study 



for A^ tending to infinity of the Laplace transform: 

L^(/x) = E^exp (-|a*(M),a) , (31) 



for arbitrary a G M'' and a positive real number /x, where {M)]\f is defined by (28) 
It can be rewritten as 



N 



L^(/x) = E,exp -^J^CMnCn , (32) 



2 



n=l 



where Ain = -^ — an(ya*al^, an is defined by (29) and ( satisfies the equation (24). 

In the sequel we will suppose that all the eigenvalues of Aq are simple and different 
from 0. Actually, it is not a real restriction, since the general case can be studied 
by using small perturbations arguments. 

Lemma 4.1. The Laplace transform -^^(/i) can be written explicitly in the follow- 
ing form: 

LUf^) = ( ( n det A J det ^jv ) , (33) 



where A„ is defined by equation (25) and 

N 



a^+i^jv = ^oJ HiA^ 8) ^" + Id2px2p ® A^)3*%. (34) 



n=l 



Here is the Kronecker product, \I'o = (Id2px2p 



2px2p ^2px2p)^ 



A = ( ^°"' ^°"'^^* ^ f35) 

^ \ fxaa* A* + ixaa*Ao%b* J ^ ^ 

and 2x2 matrices A", A2 are defined by 

Proof. The following equality can be proved by using the same arguments as those 
used in [TT] (see equations (15) and (27)): 



10 



N 



Ll{li) 



W det A„ det ^jv 



\n=l 



where ^ = ((^E^^, ^„), n > 1) is the solution of the following equation: 

with the initial condition \l'o = (Id2px2p 02px2p)- This equation can be rewritten 
as 



^i = ^LiA;i + fi^l_,Mn-iA-\ n>l, 

^l = ^i_,A-Htal + ^l_,{fiMn-iA-Htal + Al), n>l. 



Now let us denote by "$1 = a^+i^^ and ^^ 



^? 



Id 




pxp 







pxp 



pxp 

Id 



pxp 



Then 



\l/„ = (\E'^ "^D satisfies for n > 1 the following equation 
/ An' -/3„Id 



^ r 



^n— 1 



-1 



lxaa*AQ 



'■pxp 



Id 





pxp 



pxp 



-l3nAo'bb* 
fj.aa*AQ^bb* + A^ 





0. 



pxp 



\ -/3n{fiaa*AQ^) Opxp -/3„(yuaaMo ^66* + A*) 
Let TT be the following permutation of {1, ■ ■ ■ , ip} : 



^pxp 
Pn^^pxp 
-l-Clpxp 






TT U 



^5 ) 

k + 1, i = 2k + l 

p + r, i = 2r 

2p + k + l, i = 2p + 2k + l 

3p + r, i = 2r + 2p 



(37) 



where k = 0, ■ 
tation matrix 



(p — 1) and r = 1, 



p. Denote by J the correspond permu- 



Jij — "J7r(j)) ^; J — J-; ■ ■ ■ ; '*P- 



Then (pn = ^nJ satisfies the following equation: 

(Pn = <^n-l {Af, O A^ + Id2px2p ® ^2) . 

which implies that 



(3^ 



N 



"Pr 



^oJn(A®^i+id 



2px2p 



®A" 



2Jy 



n=l 



and consequently that af^^i"^}^ satisfies equality (34). 



D 
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Preparing for the asymptotic study we state the following result: 

Lemma 4.2. Let (/3„)„>i be a sequence of real numbers satisfying the condition 
(20 ). For a fixed real number a let us define a sequence of 2 x 2 matrices {S]\f{a))N>i 
such that: 

SN{a) = n f _ap^ 1 ) = n («^i + ^2), (39) 

n=l ^ ' n=l 



where A^ and A^ are defined by equation (36). Then 



1. if \a\ < 1, sup ||S'jv(a)|| < 00, 

Af>l 

2. if \a\ > 1, sup ||(5'Ar(a))^-'-|| < 00, 

Af>l 

3. if a is sufficiently small, inf trace((S'jv"'^(-))5'Ar(a)) > 0. 



Proof. The proof of assertions 1 and 2 follows directly from the estimates: 

||aA^ + A^||<l + /3^fi±^y when |a| < 1, 



a2-l 



||(aA'^ + A^)-l<l + /32(-^l, when |a| > 1. 
The proof of assertion 3 follows from the equality 

1 f a —Pn \ r^ I \ ( ^ (^t^N 



^^^^^ - r^^ I -a/3^ 1 J^^-^^^\/3^ 



for Gn{a) = 5'„^(-))5'„(a). Hence trace(GAr(0)) = -^ and condition (20) implies 



a- ■ ' ' ■■ cr^ 



that lim a"^ = TT (l + /3^) < 00 which achieves the proof. D 

n=l 

Actually, in the asymptotic study we work with a small value of fi. Note that 



for a small /i, matrix A^ defined by (35) can be represented as: Af,, = ^0 + f^H 



where 

il = ( ^0 ""* ] fj = ( ^P^P ^P^P 1 (Ar\'\ 

-^^ \ Opxp Al J \ aa* aa*AQ^bb* > ' ^ ^ 



Representation (40) implies that if the spectral radius r{i!}) < 1 then there are 
p eigenvalues of A^ such that |Ai(/i)| > 1 (in particular Aj(0), i = 1, ■ ■ ■ , p are 
the eigenvalues of A^^) and p eigenvalues of Af_i such that |Aj(/i)| < 1, j = p + 
1, ■ ■ ■ , 2p. 

12 



Lemma 4.3. Suppose that r{i)) < 1. Let us take /i 

l'm = n ("'*"' 

Then, under condition pi), 



N 



'17 

and denote by Ljy{fi): 



A,(0) 



hm m = 1, 



N^oo 



^1? 



^7v(/^) 



(41) 



(42) 



Proof. Thanks to the definition ( 25 ) of A„ the equahty 

N r 



N 



Yl det A„ = JJ 



n=l 



n=l 



:i-A 



2np_ 



1 



nr=iA.(o) 






N 



holds. Then due to equation ( |33| ) to prove ( |42| it is sufficient to check that 

1. 



hm 



detal^^i^jv 



^--K+i)'nLi[A.(/i)] 



TV 



(43) 



Diagonahzing the matrix Af_i, i.e., representing Afj, as A^ = G^D{Xi{fi))G^^ with 
a diagonal matrix D(Aj(/i)), we have also 



^^ ® A^ + Id 



2px2p 



A" 



(G^ ® Id2px2p)(^(Ai(/i)) ® A^ + Id2px2p ® ^2)(G': 



Id 



2px2p) 



This equation means that representation (34) can be rewritten as: 



a^+i^jv = *oJ(G'^ ® Id,p,2p)DiSr,i\i{fi))){G-^' ® Id2px2p)J*^S, (44) 

where D(5'Ar(Ai(/i))) is a block diagonal matrix with the block entries 5'Ar(Aj(/i)), i < 



2p defined by equation (39). Since Go is a lower triangular matrix, it follows from 



(44) that 



a^+i^jv = P^V,{SN{\{fi)))Q^ + R,V2{SN{Xj{fi)))T^, 



where 



hm P^Q^ 

fi—>-0 



Id2px2p, hm R^ = 



2px2p5 



and the block diagonal matrix Pi(S'Ar(Aj)) (respectively V2{Sn{Xj))) is such that 
|Aj(/i)| > 1 (respectively |Aj(/i)| < 1 ). 



Since detPi(S'Ar(Aj(/i))) = (c^+i) riiLii-^il/^)]^ then, by Lemma 4.2 we get 

deta^ ^^j^ 



lim 

iv^oodetr'i(^7v(Ai(/i))) 



which achieves the proof. 



D 
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The following statement plays a crucial role in the proofs. 
Lemma 4.4. Suppose that r{'d) < 1. Then under condition ([2]), for any a eW 

1 . /I 



N^oo iV 



exp I — -a*I('i9)a 



(45) 



where X('(9) is the unique solution of Lyapunov equation ^. 



Proof. It follows immediately from Lemma 4.3 that under condition (pi) the limit of 



L^{jj) does not depend on the structure of noises ^, i.e., does not depend on /3„. 
Thus, this limit is the same as for the classical i.i.d. situation, when /3„ = 0, n > 1. 

D 



Remark 4.1. It is worth mentioning that equation (41) says that 

P 7 

X]-i-ln(A,(/i))|^=o = cy*I{i9)a, 
i=i " 

where Aj(yu) are the eigenvalues of A^ such that |Aj(/i)| > 1 and X(^) is the 
solution of the Lyapunov equation ^. Of course, this equality can be proved 
independently. 



5 Proofs 



5.1 Proof of Theorem 2.1 



The statement of Theorem follows from Lemma 4.4 since (45 ) implies immediately 
that 

P^- hm 1(M)^=X(^), (46) 

and, hence also due to the central limit theorem for martingales, 

^ Mjv^A/'(0, X(^)). 



5.2 Proof of Theorem 2.2 



Due to the strong law of large numbers for martingales, in order to proof the strong 
consistency we have only to check that 



lim (M)n 



+00 a.s. 



14 



or, equivalent ly that for a one fixed constant fi > 

lim E^exp(-^{M)N 



(47) 



But in the case when p = 1 the ingredients in the right hand side of formulas (33) 



(34) with a = 1 can be given more explicitly: 

N 



Yl det A, 



N 2 



d''a 



N+li 



and 



^7V+l^JV = ^ 



ra=l 

1-A_ 



a_^-^t) + £ 



aT^aT^^^T^' 



(4J 



where the matrix S]\[{a) is defined by equation (39), 






^2 + /i + 1 ± v/(/i + (i-^)')(/u + (i + ^)') 



2i? 



are the two eigenvalues of the matrix A^j, 



X i , q I Note that — t— - = 1, 






and A+ > 1 for every /i > and -(9 G M. Equations (48) and (39) imply that for 



?9 



1-A_ 



> 1 



det "^l 



N 



a 



N+l 



and that 



Thanks to Lemma 



L_^] det U{^ 



det[Id2^2 + n{SNi^))-'SNi^ 



det S 



'ATI 



^ 



ir^A^rr^ 



det('ld2x2 + «:(5jv(^))"'5^(^) 

is uniformly bounded and separated from when /i is sufficiently large (and so 
a = -^ is sufficiently small). Since A+ > 1, we obtain that 

_ JV 

lim Lt(a) = c lim A , ^ =0. 

The uniform consistency and the uniform convergence of the moments on compacts 
K C (—1, 1) follow from the estimates (see ^15], Eq.17.51): 



E, 



N 



(M) 



N 



< (l-^2)-g^ 
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E 



(_^M„)'<(vr3^ 



Remark 5.1. It is worth mentioning that even in a stationary autoregressive 
models of order 1 with strongly dependent noises the Least Square Estimator 
^^ = ^^jI ^;-'^" is not consistent. 
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